Universal Source Codes Lecturer : Himanshu Tyagi Scribe : Sandip Sinha

• Huffman Code (optimal prefix-free code) • Shannon-Fano code • Shannon-Fano-Elias code • Arithmetic code (can handle a sequence of symbols) In general, the first three codes do not achieve the optimal rate H(X), and there are no immediate extensions of these codes to rate-optimal codes for a sequence of symbols. On the other hand, arithmetic coding is rate-optimal. However, all these schemes assume knowledge of the distribution. We will now study universal source codes. In the case of universal source codes, the distribution of the source P is unknown. We look at coding schemes for compression in several regimes: 1. Fixed length codes attaining the optimal error exponent-We have already seen this as an application of the Method of Types. 2. Variable length codes which attain: • Minimax optimality over a family of distributions • Minimax optimality for worst case individual sequence, when compared with a class of expert algorithms (for instance, all iid distributions over the alphabet) For the rest of the lecture, Q n will be an element of P(X n), i.e. a pmf on X n , Θ will refer to a family of distributions indexed by the parameter θ, and x will denote an element of X sequence of length n.